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FOREWORD 


The  Instructional  Quality  Inventory  (IQI)  was  developed  in  support  of 
Navy  Decision  Coordinating  Paper,  Education  and  Training  Development 
(NDCP-Z0108-PN) , under  subproject  P.30A,  Adaptive  Experimental  Approach  to 
Instructional  Design,  and  the  sponsorship  of  the  Director  of  Naval  Education 
and  Training  (OP-99).  The  overall  objective  of  the  subproject  is  to  develop 
an  empiricably-based  instructional  design  support  system  to  aid  developers 
in  deciding  on  instructional  alternatives  based  on  cost/benefits  and  specified 
resource  limitations.  The  purpose  of  the  IQI,  which  was  originally  called 
the  Instructional  Strategy  Diagnostic  Profile  (ISDP),  is  to  provide  quality 
control  and/or  evaluation  procedures  for  instructional  development. 

A number  of  reports  have  been  published  on  the  IQI/ISDP.  The  first  pro- 
vided an  interim  training  manual  for  the  ISDP  (NPRDC  Special  Report  77-14), 
and  the  next  three  addressed  its  empirical  and  \ "kshop  evaluations  (NPRDC 
Technical  Reports  77-25  and  77-43  and  Special  Report  78-17).  As  a result 
of  these  evaluations,  the  ISDP  was  extensively  revised  and  retitled  as 
the  IQI.  These  revisions  were  included  in  NPRDC.  Technical  Note  78-5, 
which  provided  an  interim  training  manual  for  the  IQI. 

This  report  is  the  first  in  a series  of  four  on  IQI  procedures.  The  remain- 
ing three  reports,  to  be  published  in  early  197°,  include: 

1.  An  IQI  User's  Manual,  which  will  include  a complete  description  of  all 
IQI  procedures,  and  provide  examples  of  their  use. 

2.  An  IQI  Training  Workbook,  which  will  provide  additional  examples,  along 
with  practice  on  the  101  procedures. 

3.  An  IQI  Job  Performance  Aid,  which  will  provide  a brief  version  of  each 
IQI  procedure. 

When  these  four  reports  have  been  published,  previous  training  manuals  (NPRDC 
Special  Report  77-14  and  Technical  Note  78-5)  will  have  been  superseded. 

The  IQI  is  intended  for  use  by  the  Chief  of  Naval  Education  and  Training; 
the  Chief  of  Naval  Technical  Training;  the  Chief  of  Naval  Education  and  Trainina 
Support  (specifically,  the  Instructional  Program  Development  Centers);  the 
Commander  Training  Command,  Atlantic;  the  Commander  Training  Command,  Pacific; 
and  all  other  Navy  activities  concerned  with  the  development,  revision,  or 
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SUMMARY 


Problem 


Modern  military  Instruction  Is  developed  according  to  a systematic 
method  called  Instructional  Systems  Development  (ISD).  This  method  In- 
cludes the  following  steps: 

1.  Job/task  analysis  leading  to  specification  of  training  objectives. 

2.  Development  of  tests  to  met-ure  student  progress  toward  the  objectives. 

3.  Design  of  new  Instruction  and/or  adaptation  of  existing  Instruction 
to  chleve  the  objectives. 

4.  Implementation  of  the  training  program. 

5.  Evaluation  and  feedback  for  course  maintenance. 

Various  military  activities  are  using  this  method  to  develop  or  revise 
many  of  their  training  courses.  Th*s,  It  Is  Important  that  quality  control 
and/or  evaluation  procedures  be  developed  for  use  with  ISD.  Such  procedures 
are  needed  so  that: 

1.  Quality  can  be  maintained  throughout  Instructional  design.  Thus,  errors 
made  early  In  the  development  process  will  not  be  magnified  as  development  pro- 
ceeds. 

2.  Existing  materials  can  be  evaluated  with  respect  to  newly  derived  train- 
ing objectives,  so  that  they  can  be  modified  or  revised  If  necessary. 

3.  Performance  deficiencies  of  course  graduates  can  be  traced  to  possible 
deficiencies  In  Instructional  materials. 

4.  Instructional  materials  obtained  through  contract  efforts  can  be  evaluated 
before  they  are  accepted. 

Purpose 

The  purpose  of  this  research  and  development  effort  was  to  develop  quality 
control /evaluation  procedures  for  the  three  main  products  of  an  Instructional 
development  effort;  that  Is,. objectives,  tests,  and  Instructional  materials  or 
presentations.  These  procedures  are  Intended  for  use  by  military  Instructional 
design  and  development  personnel. 

Background 

The  Instructional  Strategies  Diagnostic  Profile  (ISDP)  was  originally  developed 
by  Courseware,  Inc.  tinder  contract  to  the  Bevy  Personnel’  Research  end  Development 
Center  (Merrill,  Richards,  Schmidt,  ft  Hood,  1977).  Its  purpose  was  to  provide  a 
method  for  diagnosing  defects  In  Instructional  materials. 


Since  thst  time,  the  ISDP  procedures  have  been  extensively  evaluated 
through  workshop  trials  (Merrill,  Wood,  Baker,  Ellis,  & Wulfeck,  1977;  Merrill, 

& Wood,  1977;  Wood,  Ellis,  & Wulfeck,  1978).  As  a result  of  these  evaluations, 
the  ISDP  was  revised  and  retitled  the  Instructional  Quality  Inventory  (IQI) 
(Ellis,  Wulfeck,  Merrill,  Richards,  & Wood,  1978).  This  document  is  the  latest 
revision  of  the  IQI.  It  Is  based  on  results  of  extensive  field  tests,  and 
designed  to  be  maximally  useful  to  Instructional  design  and  development  personnel 
and  Is  tailored  to  the  needs  of  Instructional  design  activities. 

Overview 


The  current  IQI  procedures  were  designed  to  parallel  and  supplement  the 
ISD  process,  and  are  based  on  a system  for  classifying  objectives,  test  Items, 
and  Instructional  presentations  (the  three  main  products  of  Instructional 
development).  Classification  Is  determined  according  to  (1)  what  the  student 
Is  required  to  do  with  tie  information  he  learns,  and  (2)  what  type  of  Informa- 
tion the  student  Is  learning.  The  IQI  procedures  include  the  following: 

1.  Since  all  ISD  steps  depend  on  careful  specification  of  learning 
objectives,  the  first  IQI  procedure  is  to  assure  the  adequacy  of  objectives. 

This  is  done  by  classifying  each  objective,  and  judging  whetner  or  not  It 
accurately  reflects  the  Intended  student  performance  after  training. 

2.  'he  next  step  Is  to  ensure  that  tests  accurately  measure  progress 
toward  the  objectives.  This  Is  done  by  assessing  consl stency  between  test 
Items  and  their  associated  objectives.  Essentially,  each  test;  Item  must  be 
classified  In  the  same  way  as  Its  objective.  After  test  Items  and  objectives 
are  consistent,  the  adequacy  of  the  test  Items  is  assessed. 

3.  The  final  step  Is  to  ensure  that  the  Instructional  presentation  Is 
(a)  consistent  with  the  objectives  and  tests,  and  (b)  adequate  according  to 
psychological  principles  of  learning. 

Uhat  the  IQI  Is  and  Is  Not 

The  IQI  is  basically  a logical  analysis  method  that  uses  sound  principles 
of  training  and  th*  psychology  of  learning  and  instruction.  It  deals  principally 
with  cognitive  and  psychomotor  Instruction  rather  than  attitudes  and  motivation. 

The  time  and  effort  needed  to  validate  and  revise  an  Instructional  course 
or  system  can  be  groatly  reduced  by  using  IQI  analytic  procedures.  Although 
the  IQI  can  reduce  the  need  for  course  validation  using  real  students,  there 
Is  sv.111  a need  for  empirical  tryouts. 

The  IQI  Is  a method  for  product  rather  than  process  evaluation.  Regardless 
of  the  development  methodology  used  to  produce  the  products* ^that  Is,  objectives, 
tests,  or  Instruction,  the  IQI  can  he  used  to  evaluate  their  quality.  Although 
IQI  procedures  may  be  considered  during  Instructional  development,  the  IQI  Is 
Intended  to  serve  as  a supplement  to  ISD,  net  as  ft  replacement  for  It. 
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Finally , the  IQI  Is  Intended  for  use  by  people  who  are  familiar  with  ISD; 

It  cannot  be  used  by  untrained  personnel.  Also,  Its  application  depends  upon 
a good  task  analysis  or  the  availability  of  subject  matter  experts— and  pref- 
erably both.  This  Is  because  the  IQI  assumes  that  what  needs  to  be  taught 
has  already  been  determined. 

This  report  provides  an  Introduction  to  and  overview  of  the  IQI  procedures. 
It  Is  designed  to  acquaint  managers  of  Instructional  development  efforts,  in- 
struction evaluators,  and  contract  monitors  with  the  IQI.  While  It  provides 
a substantive  overview  of  the  IQI  process.  It  Is  not  a complete  IQI  training 
program. 

The  following  reports  In  the  IQI  series  will  be  available  In  early  l»7v: 

1.  A User's  Manual,  which  will  Include  a complete  description  of  oil  IQI 
procedures,  and  provide  examples  of  their  use. 

2.  A Training  Workbook,  which  will  provide  additional  examples  and  practice 
on  the  IQI  procedures. 

3.  A Job  Performance  Aid,  which  will  provide  a brief  version  of  each  IQI 
procedure. 

Conclusions 


In  previous  empirical  tests  and  In  field  workshops.  It  has  been  shown  that 
th#  IQI  is  an  affective  method  for  assuring  the  quality  of  instructional  pro- 
grams. In  addition,  there  Is  great  user  Interest  In  the  IQI;  over  300  copies 
of  previous  versions  of  the  IQI  have  been  distributed,  and  workshops  on  earlier 
versions  have  been  given  for  many  Navy  conmands.  Based  on  feedback  from  work- 
shop participants,  the  IQI  appears  to  be  a very  useful  addition  to  Navy  instruc- 
tional design  and  development  methods. 

Recommendations 

1.  This  report,  and  the  forthcoming  IQ!  user  and  training  manuals,  should 

be  published  as  NAYEDTRA  publications,  and  made  an  official  port  of  Navy  Instruc- 
tional design  end  development;  procedures. 

2.  The  IQI  should  be  considered  for  use  by  Training  Program  Coordinators 

In  the  Naval  Technical  Training  Command  (and  other  coantmis  concerned  with  train- 
ing management)  as  an  evaluation  and  quality  control  tool. 

3.  The  IQI  procedures  uhould  be  considered  for  adoption  In  the  Instructional 
Program  Development  Centers  under  the  Chief  of  Naval  Education  and  Training 
Support. 

4.  The  IQI  procedures  should  be  considered  for  use  by  technical  monitors 
of  contracts  for  training  program  as  evsluat lon/acceptance  tools. 

5.  The  IQI  procedures  should  be  considered  far  Inclusion  In  Instructor 
training  for  those  Instructors  who  will  have  curriculum  development  responsibility. 
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INSTRUCTIONAL  QUALITY  INVENTORY  PROCEDURES 


i 

THE  CLASSIFICATION  SYSTEM: 

The  following  classification  system  is  used  in  all  IQI  procedures. 

It  is  applied  to  the  three  main  parts  of  instruction:  objectives,  » 

tests,  and  instructional  presentations . 

Each  objective,  test  item,  or  piece  of  presentation,  can  be  classified 
according  to: 

1.  What  the  student  must  do,  i.e.,  the  TASK  to  be  performed,  and 

2.  The  type  of  information  the  student  must  learn,  i.e.,  the 
instructional  CONTENT. 


TASK 


CONTENT 


J 


In  the  IQI.  these  two  classification  dimensions  have  been  combined 
to  form  the  TASK/CONTENT  MATRIX. 


the  TASK  dimension: 


There  are  two  main  TASKS  a student  can  perform: 

1.  He  can  REMEMBER  information,  or 

2.  He  can  USE  the  information  to  do  something. 


REMEMBER 

USE 


EXAMPLE: 


Here  are  two  test  items: 

1.  The  symbol  for  resistor  is 


2.  Using  your  knowledge  of  electronic  theory,  what  would  happen 
in  the  circuit  shown  below  if  the  load  resistance  were 
shorted? 

These  two  test  items  differ  with  respect  to  what  the  student  is 
supposed  to  do  (TASK),  in  number  l,  the  student  has  to  REMEMBER 
something , and  in  number  2,  the  student  has  to  apply  or  USE  his 
knowledge  in  a new  situation. 
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the  CONTENT  dimension: 


There  are  five  types  of  CONTENT: 

FACT  CONCEPT  PROCEDURE  RULE  PRINCIPLE 

REMEMBER  | T 

USE 


FACTS  are  simple  associations  between  names,  objects,  symbols, 
locations , etc. 

CONCEPTS  are  categories  or  classifications  defined  by  certain 
specified  characteristics. 

PROCEDURES  consist  of  ordered  sequences  of  steps  or  operations 
performed  on  a single  object  or  in  a specific  situation. 

RULES  also  consist  of  ordered  sequences  of  operations,  but  can 
be- per formed  on  a variety  of  objects  or  in  a variety  of  situations. 

PRINCIPLES  involve  explanations  or  predictions  of  why  things 
happen  in  the  world.  That  is,  they  concern  predictions  or 
interpretations  based  on  theoretical  or  cause-effect  relationships. 


NOTE: 


Facts  can  only  be  remembered.  The  others  can  be 
remembered  or  used. 
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EXAMPLES: 

The  following  examples  illustrate  the  five  content  areas  for  the 
REMEMBER  task  level: 

1 . The  symbol  for  resistor  is . 

2.  The  student  will  list  the  names  of  the 
parts  in  the  wind  indicating  instrument. 

1.  List  the  defining  characteristics  of  a jet  pump. 

2.  The  student  will  define  the  various  kinds  of 
clouds  ( cumulus , stratus , etc.). 

1.  List  in  order  the  steps  for  cleaning  an  M-ld  rifle. 

2.  The  student  wi.ll  describe  the  procedure  for 
preparing  and  sending  a radio  message. 

1.  List  the  steps  involved  in  finding  the  rhumb- 
line course  between  two  points  on  the  earth. 

2.  The  student  will  state  the  general  rule  for 
solving  for  circuit  current , given  voltage 
and  resistance. 

1.  State  the  principles  of  electron  movement  in  a 
semiconductor  junction. 

2.  The  student  will  recall  the  reasons  why  hydraulic 
fluid  contamination  must  be  avoided. 

Facts  can  only  be  remembered , but  for  the  other  content  types , the 
student  may  be  asked  to  USE  his  knowledge  to  classify , perform , solve, 
or  predict.  The  following  are  examples  of  the  USE  task  level  for 
all  content  types  except  facts: 

USE  CONCEPT  l. 

2. 


USE  PROCEDURE  l. 

2. 

USE  RULE  l. 


USE  PRINCIPLE  l. 

2. 


Which  of  the  pumps  aboard  ship  are  jet  pumps? 

Given  photographs  of  clouds , the  student  will 

sort  them  according  to  type  (cumulus,  stratus,  etc.). 

Clean  an  M-16  rifle. 

The  student  will  prepare  and  send  a radio  message. 

Calculate  the  rhumb-line  course  from  Pearl 
Harbor  to  Long  Beach. 

Given  the  values  for  voltage  and  resistance , the 
student  will  calculate  the  au^ent  flow. 

Describe  the  theoretical  movement  of  electrons  in 
a PNP  transistor. 

The  stud.ent  will  predict  what  is  likely  to  ocnur 
if  the  landing  gear  fluid  were  oontavrinated. 


REMEMBER  FACT 


REMEMBER  CONCEPT 


REMEMBER  PROCEDURE 


REMEMBER  RULE 


REMEMBER  PRINCIPLE 
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THE  USE  LEVEL  CAN  BE  FURTHER  DIVIDED  INTO  TWO  TYPES: 

I.  USE-UNAIDED  in  which  the  student  has  no  aids  except  his  own  memory. 

2 USE-AIDED  in  which  the  student  has  a job  aid  for  performing  the  task. 

For  this  level , the  nature  of  the  aid  depends  on  th<  content  tyj>c: 


For  IISE-AIDFD  CONCEPTS  the  aid  should  consist  of  a decision 
strategy,  including  each  critical  characteristic,  and  the  decision 
to  be  made  according  to  presence  or  absence  of  that  characteristic. 
In  simple  cases,  the  aid  may  only  include  a list  of  characteristics ; 
the  decision  strategy  is  then  implied. 

For  USE-AIDED  PROCEDURES  the  aid  would  be  a list  of  steps  to  be 
performed. 


For  USE-AIDED  RULES  the  aid  would  be  at  least  a statement  of  the 
formula  or  rule  to  be  applied,  and  could  include  guidelines  for 
wheri  and  how  to  apply  it. 


EXAMPLES: 


For  USE-AIDED  PRINCIPLES  the  aid  would  also  be  at  least  a statement 
of  the  principle,  and  could  include  guidelines  for  when  and  how 
to  apply  it. 

rais  page  is  JU.UTY 
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USE-AIDED:  A pilot’s  preflight  checklist  is  a USE-AIDED 

procedure.  The  pilot  does  not  have  to  remember  the 
steps  or  their  order  because  they  are  on  the  check- 
list. The  pilot  does  need  to  perform  the  steps 
correctly . 


USE-UNAIDED:  "The  student  will  field-strip  an  M-16  riftle.  " 

Here,  the  student  must  remember  the  steps  in  the 
correct  order,  and  perform  them  correctly. 


In  summary,  the  REMEMBER  level  involves  "pure"  remembering, 

the  USE-UNAIDED  level  involves  remembering  what  is  to  be  used, 
and  then  using  it,  and 

the  USE-AIDED  level  involves  "pure"  using. 
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the  entire  TASK  / CONTENT  MATRIX  is  shown  below: 


FACT 

RECALL  OR 
RECOGNIZE 
NAMES> 

PARTS, 

DATES/ 

PLACES/  ETC. 


CONCEPT 

REMEMBER 
CHARACTERIS- 
TICS/ OR 
CLASSIFY  OB- 
JECTS/ EVENTS 
OR  IDEAS  AC- 
CORDING TO 
CHARACTERISTICS 


PROCEDURE 

SEQUENCE  OF 
STEPS  REMEM- 
BERED OR  USED 
IN  A SINGLE 
SITUATION  OR 
ON  A SINGLE 
PIECE  OF 
EQUI  FflENT 


RULE 

REMEMBER  OR 
USE  A SEQUENCE 
OF  STEPS  WHICH 
APPLY  ACCROSS 
SITUATIONS 
OR  ACROSS 
EQUIPMENTS 


PRINCIPLE 

REMEMBER/  OR 
INTERPRET  / 
PREDICT/  WHY 
OR  HOW  THINGS 
HAPPEN,  OR 
CAUSE-EFFECT 
RELATIONSHIPS 


REMEMBER  - recall  or 

RECOGNIZE  FACTS/  CON- 
CEPT DEFINITIONS,  STEPS 
OF  PROCEDURES  OR  RULES, 
STATEMENTS  OF  PRINCIPLE 

USE-UNAIDED  - tasks  which  require 

CLASSIFYING,  PERFORMING  A PROCEDURE, 
USING  A RULE,  EXPLAINING  OR  PREDICTING 
WITH  NO  A I DC  EXCEPT  MEMORY. 

USE- AIDED  - same  as  use- unaided, 

EXCEPT  JOB  AIDS  ARE  AVAILABLE. 

Any  objective,  test  item,  or  piece  of  instruction  will  be 
classifiable  in  one  and  only  one  cell  of  the  matrix  above. 

This  matrix  is  used  in  all  IQI  steps. 
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OBJECTIVE  ADEQUACY: 


IBIS  PASS  IS  BEST  QUALITY  FRACIiCAM** 

HUM  OOPY  imytlSHBD  TO  DDft 

T’fee  first  step  in  the  TQI  procedure  corresponds  with  the  development 
of  training  objectives.  The  procedure  described  below  is  used  to 
determine  if  each  objective  is  adequate  for  further  instructional 
development. 

Objectives  are  ADEQUATE  if  they  satisfy  three  general  criteria: 

1.  Is  the  objective  CORRECTLY  STATED?  Does  the  objective  include  statements 
of  actions  the  student  is  to  perform  after  training , the  conditions  under 
which  the  performance  is  expected , and  the  standards  which  the  performance 
must  meet?  If  even  one  of  these  parts  is  missing,  the  objective  is 
inadequate  because  training  for  it  cannot  be  designed  or  evaluated. 

EXAMPLE:  Inadequate  objective:  " The  student  will  prepare  a standard 

Navy  message.  " This  is  inadequate  because  it  does  not  specify 
either  the  conditions  ( given  a typewriter?  TTY?)  or  the 
standards  (how  fast  and  how  many  errors). 

2.  Is  the  objective  CLASSIFIABLE  on  the  task/content  matrix?  If  the  ob- 
jective cannot  be  classified , this  means  that  the  action  the  student 
is  to  perform  is  not  stated  clearly  enough  so  that  we  know  what  the 
student  is  to  do.  Training  cannot  be  designed  or  evaluated  in  these 
circumstance 8. 

EXAMPLE:  The  objective  "The  student  will  learn  repair  procedures  for 
the  XYZ  radar  set"  is  not  classifiable.  It  is  not  clear 
whether  the  student  should  remember  the  procedures  or  actually 
use  them. 

3.  Is  the  "intent"  of  the  objective  APPROPRIATE  for  the  purpose  of  the 
course?  The  actions,  conditions,  and  standards  specified  in  the 
objective  should  be  as  close  as  possible  to  the  actions,  conditions , 
and  standards  of  the  task  to  be  performed  on  the  job  after  training. 

In  addition,  it  is  assumed  that  the  ultimate  " intent " of  any 
training  program  is  to  teach  the  student  how  to  do  something  (i.e.  ,USE 
level).  Therefore,  there  must  be  a USF.-level  objective  for  each  REMEMBER 
objective.  (Facts  are  a special  case:  Although  facts  are  not  used,  they 
often  must  be  taught  to  provide  a knowledge  base  for  a later  use-level 
task.  Therefore , in  order  to  justify  teaching  facte,  they  must  support 
some  use-level  objective.) 

Conversely,  USE-UNAIDED  tasks  should  be  taught  at  the  REMEMBER 
level  before  being  taught  at  the  USE  level.  'Therefore,  just  as  every 
REMEMBER  objective  should  have  a corresponding  USE  objective,  every  USE 
objective  should  have  a previous  REMEMBER  objective. 

EXAMPLE:  The  objective  "The  student  will  identify  the  connection  of  a 

voltmeter  to  measure  the  voltage  across  a component  by  selecting 
an  illustration " is  not  appropriate  for  the  intent  of  the  course. 
The  student  will  not  see  possible  illustrations  of  connections 
on  the  job,  but  will  be  required  to  set  up  the  connection , thus 
the  action  should  be  revised. 
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TEST  CONSISTENCY  and  ADEQUACY: 


Once  objectives  are  adequate,  test  items  can  be  developed.  The  next 
IQT  step  is  the  quality  control  step  for  test  development.  This  step 
involves  determining  whether  test  items  are  CONSISTENT  with  objectives, 
and  whether  each  item  is  ADEQUATE. 

A test  item  is  CONSISTENT  with  its  objective  if: 

1.  The  ACTION  (TASK/CONTENT  level)  of  the  test  Item  is  the  same  as  that 
of  the  objective. 

2.  The  CONDITIONS  under  which  the  item  is  administered  are  as  close  as 
possible  to  those  of  the  objective. 

3.  The  STANDARDS  in  the  test  item,  or  the  STANDARDS  for  scoring  the  item, 
are  as  close  as  possible  to  the  standards  in  the  objective. 


EXAMPLE:  Objective:  Given  the  necessary  tools  and  an  operator’s 

manual,  the  student  will  set  up  and  operate 
a double-acting  reciprocating  pump,  in  five 
minutes  and  according  to  the  manual  specifications. 

Inconsistent  test  item:  ", List  the  steps  of  procedure  for 
starting,  operating , and  stopping  a double- 
acting  reciprocating  pump . " 

THIS  PAGE  IS  BEST  QUALITY  PRACTICABLE 

mag  giipfl  y nnfl  - This  test  item  is  inconsistent,  because 

^ - it8  TASK/ CONTENT  is  REMEMBER-PROCEDURE 

instead  of  USE-AIDED- PROCEDURE.  Notice 
that  the  action  the  student  is  to  perform 
in  the  test  is  not  the  same  as  the  action 
required  in  the  objective. 

Consistent  test  item:  "Use  the  operator's  manual,  and 

necessary  tools  to  set  up  and  operate  a double- 
acting reciprocating  pump.  You  will  pass  this 
test  if  you  complete  this  task  within  S minutes, 
in  accordance  witn  the  manual  specifications.  " 

This  test  item  is  consistent  with  the 
objective.  Notice,  however,  that  if 
either  the  conditions  or  grading  standards 
had  been  left  out , the  item  would  have 
been  inconsistent. 


4.  Tha  FORMAT  of  the  test  item  is  appropriate  for  the  TASK/CONTENT 
level . 

EXAMPLE:  " The  steps  in  the  procedure  for  operating  a 

jet  pump  are  listed  below.  Arrange  them  in 
the  correct  order. " 


This  is  an  inappropriate  format  for 
REMEMBER- PROCEDURE  because  the  student 
doesn't  have  to  remember  the  procedure,  only 
recognize  the  steps. 


Note:  Recognition  items  (multiple  choice , matching , true- false)  are 
usually  NOT  appropriate  test  formats  for  REMEMBER  level  objectives. 

This  is  because  these  items  do  not  reflect  typical  job-performance 
requirements. 

Multiple  choice,  matching , and  true-false  items  are  appropriate 
for  concept  recognition,  and  can  be  appropriate  for  USE  level  objectives 
if  they  arc  carefully  desigtxed.  However,  for  USE  level  objectives, 

" hands-on " performance  tests  are  usually  most  appropriate. 

THIS  PAGE  IS  BKST  QUALIfi 

tfBOMCOP*  lUMXSHH)  XOUCO ' 


After  a test  item  is  consistent  with  its  objective,  the  test  item  is 
ADEQUATE  if: 


1.  The  item  is  clear  and  unambiguous. 

2.  The  item  does  not  give  away  its  own  answer  or  the  answer  to  any 
other  item  on  the  test. 

3.  The  item  is  well  constructed. 

4.  Other  adequacy  concerns  covered  in  the  IQI  manual  are  met. 

EXAMPLE:  " Which  of  the  following  ..."  is  ambiguous 

because  it  does  not  say  "choose  all  that  apply" 
or  "choose  the  best  . . . . " 


PRESENTATION  CONSISTENCY: 

At  this  point  in  the  IQI  process,  objectives  are.  adequate,  test  items 
are  consistent  with  objectives,  and  test  items  are  adequate.  The 
next  irtstructional  design  step  is  to  prepare  the  instructional  materials 
or  presentations.  The  next  IQI  step  is  to  insure  that  the  presentations 
are  CONSISTENT  with  the  objectives  and  test  items,  and  ai'e  ADEQUATE. 

In  the  previous  section,  determining  test-objective  consistency  involved 
comparing  each  test  item  with  its  related  objective.  Determining 
PRESENTATION  CONSISTENCY,  on  the  other  >iand,  involves  checking  whether 
or  not  each  of  the  INSTRUCTIONAL  COMPONENTS  required  for  a given 
objective-test  item  is  present.  There  are  different  types  of  instruc- 
tional components.  In  order  to  insure  consistency,  the  appropriate 
components  must  be  present  for  each  TASK/CONTENT  level.  Not  all 
task/ content  levels  require  all  components. 

The  Instructional  PRESENTATION  COMPONENTS  are: 


1.  STATEMENT : The  instruction  tells  the  student  something  he  must  learn. 


2.  EXAMPLES:  The  instruction  shows  the  USE  of  content  ( concept , procedure, 
rule,  or  principle). 


3.  PRACTICE: 


The  student  practices  REMEMBERING  or  USING  the  content,  and 
is  given  feedback. 


—li'.ris 
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PRESENTATION  COMPONENTS : IBIS  PAfflE  IS  PEST 

TOO*  OOPYFURKISHEDIODD.CS  •*-' 

STATEMENT  Component:  The  instruction  presents  a statement  of  a fact, 

a concept  definition,  the  steps  of  a procedure  or 
rule,  or  a statement  of  a principle. 

EXAMPLES:  1.  " The  characteristics  of  a typical  jet  pump  include " 

(concept  definition) . 

2.  "The  procedure  for  changing  a gasket  in  a check  valve  is  . . . 
(procedure  definition) . 

3.  "To  determine  voltage , multiply  current  by  resistance. " 
(rule  statement) . 


EXAMPLE  Component:  The  student  is  told  or  shown  how  a statement  of  a 

concept,  procedure,  rule,  or  principle  applies  in  a 
specific  case. 


EXAMPLES:  1.  " The  XXYZ  pump  is  a double-acting  reciprocating  pump 
because  it  has  the  particular  characteristics  noted 
on  the  diagram  below."  (concept  example). 

2.  "I^t's  see  how  OHM'S  LAW  applies  in  a specific  case " 

(rule  example). 

3.  "The  Navy's  victory  at  Midway  in  World  War  IT  illustrates 
the  value  of  cryptologic  intelligence  because... " 
(principle  example). 


PRACTICE  REMEMBERING  Component:  The  student  is  asked  to  supply  part  or  all 

of  a fact  statement,  concept  definition,  the  steps 
of  a procedure  or  rule,  or  the  statement  of  a principle. 
The  student  is  given  FEEDBACK  about  the  correctness  of 
his  answer. 


EXAMPLES:  1.  "The  father  of  our  country  is ?"  (Fact) 

2.  "List  in  order  the  steps  of  procedure  for  ...."  (Procedure) 

PRACTICE  USING  Component:  The  student  is  asked  to  use  a concept  definition, 

procedure,  rule,  or  principle  on  a specific  case 
to  which  it  applies,  and  is  given  FEEDBACK  about 
the  quality  of  his  performance. 


EXAMPLES:  1.  " Classify  the  following  Lofargrams."  (concept) 


2.  "Using  the  procedure  in  the  tech,  manual,  dissassemble 
the  . ( procedure ) 


3.  "Solve  the  following  circuit  problems...."  (rule) 


4.  "Predict  the  effect  (sociological  and  psychological)  when 
women  are  assigned  to  Navy  ships."  (principle). 
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For  CONSISTENCY,  different  components  are  required  for  different  task  levels: 


Por 

thfc  REMEMBER  level: 

a STATEMENT 

(no  example) 

PRACTICE 

REMEMBERING. 

For 

the  USE-UNAIDED  level: 

a STATEMENT 

EXAMPLES 

PRACTICE 

(or  a review 

(at  least 

USING. 

of  the  state- 
ment. ) 

one) . 

For 

the  USE-AIDFD  level: 

(The  aid 

EXAMPLES 

PRACTICE 

takes  the 

WITH  AID. 

USING  WITH 

place  of  the 
statement. ) 

AID. 

These  required  components  apply  across  all  content  types  (facts, 
concepts , procedures,  rules,  and  principles)  for  REMEMBERING, 
and  all  except  facts  for  USING . For  example,  if  the  objective  and 
test  item  called  for  the  student  to  remember  a fact,  then  the  in- 
struction must  contain  a statement  of  the  fact,  to  be  remeni>ered , 
and  at  least  one  practice-remembering  item  with  feedback.  No  ex- 
ample is  required,  because  it  would  be  redundant  with  the  statement. 


THIS  PAGE 
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CONSISTENCY  also  requires  that  each  required  component  ineet  the 
following  criteria: 

1 . STATEMENTS  must  be  COMPLETE. 

2.  EXAMPLES  must  show  application  of  the  complete  content. 

3.  EXAMPLES  must  match  or  reflect  the  conditions  and  standards  required 
of  the  objective  and  the  test  as  closely  as  possible. 

4.  PRACTICE  must  include  FEEDBACK. 

5.  PRACTICE  must  be  of  the  same  task/content  level  as  the  test  item 
and  objective. 

6.  PRACTICE  must  match  or  reflect  the  conditions  and  standards  required 
of  the  objective  and  the  test  as  closely  as  possible,  or  be  designed 
to  help  the  student  gradually  learn  the  final  task. 

Most  of  the  requirements  above  are  probably  obvious , but  some  are 
compliaated.  COMPLETENESS , for  example y requires  different  pre- 
scriptions for  different  content  types: 

For  a CONCEPT:  "complete"  means  that  all  the  critical  charac- 
teristics of  the  concept 3 and  their  combination , ore  given. 

For  a PROCEDURE:  " complete " means  that  all  the  steps  of  the 
procedure  are  given  in  the  proper  order. 

For  a RULE:  " complete " means  that  all  the  steps  of  the  rule 
are  given  in  the  proper  order. 

For  a PRINCIPLE:  "complete"  means  that  all  the  pre-  and  post- 
conditionsj actions3  processes f causes , effects , and  results  are 
stated , arid  the  relationship  between  them  is  clearly  stated. 


PRESENTATION  ADEQUACY 

Once  all  the  required  instructional  components  are  present , and  each 
of  these  components  meets  all  of  the  consistency  criteria , t,ie 
ADEQUACY  of  the  presentation  can  be  assessed.  This  is  done  by  checking 
each  instructional  component  ( statement , examples , practice)  for 
certain  characteristics. 

A STATEMENT  Is  ADEQUATE  if  it  meets  the  following  criteria: 

1.  The  statement  must  be  SEPARATED  from  the  rest  of  the  instruction. 

This  helps  the  student  find  the  main  idea.  When  the  statement  is 

separated,  the  key  points  stand  out , and  are  not  buried  in  the 

presentation.  There  are  several  ways  to  accomplish  this  goal: 

a.  Set  off  the  statement  with  boxes. 

b.  'Use  different  color. 

a.  Use  a different  type,  or  underline. 

d.  Place  on  a separate  page , or  in  a special  place  on  the  page. 

e.  For  audio  or  movies,  pause  before  giving  the  statement. 

2.  The  c fcatement  muse  be  IDENTIFIED.  After  the  statement  is  separated , 
the  student  should  be  told  what  it  is.  This  permits  the  student's 
attention  to  be  focused  on  the  key  points  and  their  application,  rather 
than  the  student  trying  to  become  generally  familiar  with  everything  in 
the  instruction.  One  way  to  identify  a statement  is  to  use  the  word 
"statement.  " Other  more  conten4--  riented  words  ore  even  more  helpful: 

definition  procedure  for the  principle  of 

Mai):  Idea:  Key  Point:  General  rule: 


(Here,  the  statement 
is  separated  by  the 
box , and  identified. ) 

3.  In  addition  to  the  statement,  the  presentation  should  include  something 
to  help  the  student  better  understand  and  remember  the  statement. 

Methods  o,i  providing  this  help  include: 


EXAMPLE: 


a. 

b. 


c. 

d. 


e. 

f- 


EXAMPLE: 


Giving  a MNEMONIC  (memory  trick). 

Giving  a general  example  of  how  the  statement  oa.%  be  used. 
Explaining  why  the  statement  is  important . 

Explaining  how  it  came  about , how  it  fits  in  the  course,  or 
haw  it  relates  to  something  the  student  already  knows. 
Explaining  some  of  the  terms  in  the  statement. 

Representing  the  statement  with  pictures , symbols , flowcharts, 
tables,  to. 

The  foccowing  figure  can  be  a helpful  memory 
device  for  Ohm's.  It  will  help  you  remember 
it  so  you  can  use  it  later  an. 


4.  Other  adequacy  covered  in  the  IQI  manual  are  met. 
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EXAMPLES  are  ADEQUATE  if  they  meet  the  following  criteria: 

1.  EXAMPLES  must  be  SEPARATED  and  IDENTIFIED. 

2.  EXAMPLES  must  include  some  type  of  help. 

3.  EXAMPLES  should  range  from  "easy"  to  "hard." 

4.  EXAMPLES  should  be  representative  of  the  job  the  student  will  do 
after  training. 

5.  There  should  be  enough  examples  to  cover  the  content  area  adequately. 

6.  EXAMPLES  should  clearly  show  why  common  errors  are  wrong. 

The  criteria  are  generally  selp- explanatory . SEPARATED  and  IDENTIFIED 
are  the  same  as  for  statements , and  points  3 to  6 need  no  further 
explanation.  The  second  criterion , HELP,  is  applied  in  different 
ways  for  different  content  types.  Some  types  of  HELP  for  each  content 
type  are  given  below: 

HELPS  for  CONCEPTS:  Highlight  the  critical  characteristics  of  an 

example. 

Explain  why  or  why  not  something  is  classified 
as  a member  of  a concept. 

Show  the  use  of  a checklist  or  heuristic  to 
help  classify. 

Simplify  early  examples,  e.g.,use  live-drawings 
instead  of  complicated  photographs. 

HELPS  for  PROCEDURES  or  RULES:  Explain  why  each  step  is  done. 

Explain  why  each  step  is  important. 
Give  additional  information  about 
how  to  perform  the  task. 

Give  additional  information  about  how 
to  know  if  you've  done  it  wrong. 
Give  flowcharts,  tables,  etc. 

HELPS  for  PRINCIPLES:  Highlight  important  features. 

Simplify  the  relevant  information  from  the 
case  3tudy  in  which  it  is  embedded. 

Use  logical  representations  of  the  IF-THEN 
relationships. 

Give  additional  information  about  how  the 
principle  applies,  or  why  it  doesn’t. 

Give  hints  as  to  how  to  analyse  problems. 

Other  adequacy  concerns  are  given  In  the  1QI  manual. 
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PRACTICE  i Ltms  are  ADEQUATE  if  they  meet  the  following  criteria; 

1.  The  PRACTICE  section  must  be  SEPARATE  and  IDENTIFIED. 

2.  The  PRACTICE  Items  must  be  free  of  hints  that  wouldn't  be  present 
in  the  test  or  on  the  job. 

3.  The  PRACTICE  items  should  have  the  same  format  as  the  format  of 
the  test  items. 

4.  The  PRACTICE  items  should  range  from  easy  to  hard. 

5.  The  PRACTICE  items  should  be  typical  of  the  job  to  be  performed 

after  training. 

6.  The  PRACTICE  items  should  include  the  opportunity  for  common  errors. 

7.  The  FEEDBACK  must  also  be  SEPARATED  and  IDENTIFIED  for  each  practice 

item. 

8.  The  FEEDBACK  should  include  help  (similar  to  that  for  examples). 

(As  a bare  minimum , the  FEEDBACK  should  direct  the  student  back  to 
where  the  instruction  was  originally  presented.  However , it  is 
better  to  have  a new  brief  presentation , because  if  the  student 
got  the  praatiae  wrong , the  original  presentation  didn't  help  enough.) 


The  criteria  are  also  self-explanatory. 

Other  adequacy  concerns  are  given  .n  the  IQI  manual. 


W AMPLE: 


The  next  cxanpd.e  uhews  an  instructional  presentation  which 
violates  many  of  the  adequacy  criteria  described  above. 

This  exam$)le  is  followed  by  a more  adequate  presentation  of 
the  same  subject  matter. 


INADEQUATE  PRESENTATION  on  the  principles  of  operation  of  an  alarm  circuit: 


The  alarm  circuit  senses  extremely  high  temperatures.  When  an  j 

extreme  steam  temperature  occurs  (which  is  a very  dangerous  condition  ( 

that  may  have  adverse  consequences  for  a ship  and  her  crew),  the  sensing  < 

switch  contacts  close  .thus  shunting  the  resistor.  The  decreased  resis-  1 

tance  in  the  circuit,  according  to  OHM'S  LAW  ( E= I R) , causes  an  increase 
in  current  flow  in  the  circuit,  which  is  enough  to  operate  the  alarm 
relay.  The  relay  is  designed  to  operate  at  a current  flow  above  that 
normally  found  in  the  circuit.  OHM'S  LAW  states  that  with  voltage 
constant,  a decrease  in  resistance  in  the  circuit  must  be  accompanied 
by  an  increase  in  current  flow.  The  contacts  of  the  alarm  relay  then 
close  to  actuate  the  audible  alarm  device,  which  may  consist  of  a warning 
bell  with  an  electrically  operated  clapper,  or  an  H254  resonated  horn 
assembly.  Both  of  these  produce  extremely  loud  signals  so  they  can 
overcome  normal  ambient  noise  levels. 


* to  external  audible  alarm  device. 

Why  is  it  important  that  the  alarm  circuit  be  operational  at  all  times? 
Remember  what  hot  steam  can  do  to  ships  and  sailors. 


The  example  above  is  inadequate  in  several  ways.  First , the  principle 
of  operation  of  the  circuit  is  not  separated  or  identified.  How  is  the 
student  to  know  what  to  learn  from  this  presentation?  Second,  the 
presentation  is  cluttered  urith  a lot  of  other  "nice  to  know"  information 
that  really  doesn't  help.  If  helps  were  included,  they  should  aid 
remembering  or-  understanding  the  principles  of  operation  of  the  circuit. 
Also,  the  practice  is  not  separate  or  identified,  there  is  no  feedback, 
and  the  practice  really  has  nothing  to  do  with  remembering  the  principle. 

The  next  page  shows  a more  adequate  presentation. 
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MORE'  ADEQUATE:  I 'RESENT AT  W\  on  the  principle®  of  operation  of  an  a turn  circuit: 


USING  TIE  IQ! : 


The  IQI  is  designed  for  Quality  Control  during  any  objectives-based 
instructional  development  process. 

There  are  four  documents  that  comprise  the  IQI: 

1.  Introduction  and  Overview  (This  document) 

2.  User's  Manual  (contains  all  IQI  procedures,  and  examples 

of  their  use) 

3.  Workbook  (contains  practice  on  all  IQI  procedures,  with 

feedback) 

4.  Job  Performance  Aid  (short  version  of  all  prL  edures) 

To  facilitate  use  of  the  IQI  procedures,  the  User's  Manual,  Workbook, 
and  JPA  were  designed  to  include  three  quality  control  forms:  The 
first  form  assesses  objective  adequacy,  and  the  second  is  used  to 
determine  test  consistency  and  test  adequacy,  and  the  third  evaluates 
objective-presentation  consistency  and  presentation  dequacy.  The 
suggested  use  of  these  forms  is  as  follows: 

1.  Either  during,  or  immediately  after,  the  development  or 
objectives  in  instructional  development,  use  the  objective 
adequacy  form  to  assess  the  adequacy  of  each  objective. 

Any  required  revisions  should  be  made  before  instructional 
development  proceeds. 

2.  As  test  items  are  developed  for  each  objective,  they  should  be 
checked  for  consistency  with  objectives,  and  adequacy,  using 
the  second  form. 

3.  As  new  instructional  materials  are  developed,  or  as  existing 
materials  are  adopted,  they  are  .checked  for  consistency  with 
objectives,  and  adequacy,  using  the  third  form.  Required 
revisions  to  materials  and  tests  are  made  before  they  are 
subjected  to  individual  or  small -group  try-outs. 

FOR  MORE  INFORMATION: 

For  more  information  on  the  IQI,  contact:  phone: 

Navy  Personnel  Research  and  Development  Center  (714)  225-7121 

Code  P304 

San  Diego,  CA  92152  AUTOVON  933-7121 

7140 
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1 Officer  in  Charge,  Central  Test  Site  for  Personnel  and  Training  Evaluation  Program 
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Research  (AFSC) 
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Secretary  Treasurer,  U.S.  Naval  Institute 
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